Neuro-Evolution for Multi-Agent Policy Transfer in RoboCup Keep-Away

An objective of transfer learning is to improve and speedup learning on target tasks after training on a different, but related source tasks. This research is a study of comparative Neuro-Evolution (NE) methods for transferring evolved multi-agent policies (behaviors) between multi-agent tasks of varying complexity. The efficacy of five variants of two NE methods are compared for multi-agent policy transfer. The NE method variants include using the original versions (search directed by a fitness function), behavioural and genotypic diversity based search to replace objective based search (fitness functions) as well as hybrid objective and diversity (behavioral and genotypic) maintenance based search approaches. The goal of testing these variants to direct policy search is to ascertain an appropriate method for boosting the task performance of transferred multi-agent behaviours. Results indicate that an indirect encoding NE method using hybridized objective based search and behavioral diversity maintenance yields significantly improved task performance for policy transfer between multi-agent tasks of increasing complexity. Comparatively, NE methods not using behavioral diversity maintenance to direct policy search performed relatively poorly in terms of efficiency (evolution times) and quality of solutions in target tasks

Hybridizing Novelty Search for Transfer Learning

This study investigates the impact of genotypic and behavioral diversity maintenance methods on controller evolution in multi-robot (RoboCup keep-away soccer) tasks. The focus is to examine the impact of these methods on the transfer learning of behaviors, first evolved in a source task before being transferred for further evolution in different but related target tasks. The goal is to ascertain an appropriate controller design (NE: NeuroEvolution) method for facilitating improved effectiveness given policy transfer between source and target tasks. Effectiveness is defined as the average task performance of transferred behaviors. The study comparatively tests and evaluates the efficacy of coupling policy transfer with several NE variants. Results indicate a hybrid of behavioral diversity maintenance and objective-based search yields significantly improved effectiveness for evolved behaviors across increasingly complex target tasks. Results also highlight the efficacy of coupling policy transfer with the hybrid of behavioral diversity maintenance and objective based search in order to address bootstrapping and deception problems endemic to complex tasks

Multi-Agent Behavior-Based Policy Transfer

A key objective of transfer learning is to improve and speedup learning on a target task after training on a different, but related, source task. This study presents a neuro-evolution method that transfers evolved policies within multi-agent tasks of varying degrees of complexity. The method incorporates behavioral diversity (novelty) search as a means to boost the task performance of transferred policies (multi-agent behaviors). Results indicate that transferred evolved multi-agent behaviors are significantly improved in more complex tasks when adapted using behavioral diversity. Comparatively, behaviors that do not use behavioral diversity to further adapt transferred behaviors, perform relatively poorly in terms of adaptation times and quality of solutions in target tasks. Also, in support of previous work, both policy transfer methods (with and without behavioral diversity adaptation), out-perform behaviors evolved in target tasks without transfer learning

Neuro-evolution behavior transfer for collective behavior tasks

The design of effective, robust and autonomous controllers for multi-agent and multi-robot systems is a long-standing problem in the fields of computational intelligence and robotics. Whilst nature-inspired problem-solving techniques such as reinforcement learning (RL) and evolutionary algorithms (EA) are often used to adapt controllers for solving such tasks, the complexity of such tasks increases with the addition of more agents (or robots) in difficult environments. This is due to specific issues related to task complexity, such as the curse of dimensionality and bootstrapping problems. Despite an increasing attempt over the last decade to incorporate behavior (knowledge) transfer in machine learning so that relevant behavior acquired in previous learning experiences can be used to boost task performance in complex tasks, using evolutionary algorithms to facilitate behavior transfer (especially multi-agent behavior transfer) has received little attention. It remains unclear how behavior transfer addresses issues such as the bootstrapping problem in complex multi-agent tasks (for example, RoboCup soccer). This thesis seeks to investigate and establish the essential features constituting effective and efficient evolutionary search to augment behavior transfer for boosting the quality of evolved behaviors across increasingly complex tasks. Experimental results indicate a hybrid of objective-based search and behavioral diversity maintenance in evolutionary controller design coupled with behavior transfer yields evolved behaviors of significantly high quality across increasingly complex multi-agent tasks. The evolutionary controller design method thus addresses the bootstrapping task for the given range of multi-agent tasks, whilst comparative controller design methods yield scant performance results

Evolutionary Policy Transfer and Search Methods for Boosting Behavior Quality: RoboCup Keep-Away Case Study

This study evaluates various evolutionary search methods to direct neural controller evolution in company with policy (behavior) transfer across increasingly complex collective robotic (RoboCup keep-away) tasks. Robot behaviors are first evolved in a source task and then transferred for further evolution to more complex target tasks. Evolutionary search methods tested include objective-based search (fitness function), behavioral and genotypic diversity maintenance, and hybrids of such diversity maintenance and objective-based search. Evolved behavior quality is evaluated according to effectiveness and efficiency. Effectiveness is the average task performance of transferred and evolved behaviors, where task performance is the average time the ball is controlled by a keeper team. Efficiency is the average number of generations taken for the fittest evolved behaviors to reach a minimum task performance threshold given policy transfer. Results indicate that policy transfer coupled with hybridized evolution (behavioral diversity maintenance and objective-based search) addresses the bootstrapping problem for increasingly complex keep-away tasks. That is, this hybrid method (coupled with policy transfer) evolves behaviors that could not otherwise be evolved. Also, this hybrid evolutionary search was demonstrated as consistently evolving topologically simple neural controllers that elicited high-quality behaviors